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Field of the Invention 

This invention concerns an arrangement and a method 
for controlling an electronic device. 
Backqround _o_ f the Invent ion 

The Applicant's Swedish Patent No. 9604008-4 dis- 
closes a reading pen which can be used for recording text 
by imaging this. The reading pen has an optical sensor 
which captures a number of images, with partially over- 
lapping content, of the text which is to be recorded. In 
addition, the reading pen has a processor which puts 
together the images, identifies, the text in the images 
and stores it in character-coded format. The text can 
then be transferred to a computer with which the reading 

pen communicates . 

in addition, the Applicant's Swedish Patent No. 
9803456-4 discloses an optical mouse which is arranged 
to position a cursor on a display of a computer. 

The reading pen and the optical mouse are two exam- 
ples of input units which can be used to input informa- 
tion xnto a computer or to control the function of a com- 
puter. Another example of an input unit is a keyboard, 
which can be used both to enter information and to con- 
trol the computer by means of various keyboard commands. 

A user does not want to have to change between dif- 
ferent input units in order to be able to carry out dif- 
ferent functions. Therefore it is a general requirement 
that each input unit is to be able to be used for as many 



different functions as possible and in as flexible and 
simple a way as possible for the user. 
Summary of th e Invention 

An object of this invention is therefore to provide 
an arrangement for controlling an electronic device which 
is simple for a user to use and which makes possible a 
combination with other input unit functions. 

This object is achieved by an arrangement according 

to claim 1. 

According to a first aspect, this invention concerns 
an arrangement for controlling an electronic devxce com- 
prising an input unit with an optical sensor for record- 
ing text by xmaging this by means of at least one image, 
and signal -processing means for identifying predeter- 
mined information in the imaged text and for controlling 
the electronic device dependent upon said predetermined 
information. 

According to the invention the arrangement for con- 
trolling an electronic device is based on an input unit . 
function known per se, that is recording of text, which,, 
however, is modified by making the signallprocessing 
means identify predetermined information in the imaged 
text, unlike the previously known input unit function, 
the content of the recorded text is interpreted instead 
of the text just being stored. A user can thus control 
the electronic device on the basis of text which is read 
using the input unit. The text can be predetermined com- 
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r„andB «hich are printed or written on a sheet of paper 
or so™e other product. The text can also be text which is 
used in specific applications in the electronic device, 
3uch as addresses. The content in these addresses varies, 
but the arrangement can recognize predetermined informa- 
tion, for example certain characters, which make it pos- 
sible for the arrangement tc interpret what the text 
refers to. 

The arrangement according to the invention is simple 
for the user to use, as the user just needs to pass the 
input unit across the text or place it on the text in 
order to control the electronic device. In addition the 
arrangement can easily be combined with other input unit 
functions which are based on recording of images by an 

optical sensor. 

The signal-processing means are preferably realized 
as software. However, they can consist of a specially 
adapted hardware circuit, for example an ASIC, or some 
suitable combination of software and hardware. 

The electronic device which is controlled by means 
of the arrangement can be a computer, a mobile telephone, 
a PDA or some similar electronic device. 

The optical sensor can be a line sensor, but is pre- 
ferably an area sensor. 

in a preferred embodiment, the predetermined infor- 
mation consists of at least one predetermined character 
and the signal -processing means are arranged to generate 
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a command associated with said character for controlling 
the electronic device. The signal -processing means can be 
arranged to identify special characters such as ®, or 
predetermined words, word combinations or character com- 
binations. Each such character, word or combination can 
be associated with a special command. The command can be 
a command to the electronic device at system or applica- 
tion level. It can emulate a keyboard command. It can 
also be a special command, for example a user-defined 
command . 

The arrangement can advantageously comprise a read- 
ing pen function. The same arrangement can then be used 
by the user both for controlling the electronic device 
and for inputting text into it. The reading pen function 
can advantageously be realized in the same way as in the 
Applicant's above-mentioned patent. Other realizations 
are, however, also possible. 

In an advantageous embodiment, as an alternative to 
or a supplement to the reading pen, the arrangement can 
comprise an optical mouse function for controlling a 
cursor on a display on the electronic device. The user 
can then carry out mouse functions using the arrangement 
and also control the electronic device by recording text. 
The mouse function can advantageously be realized in the 
same way as in the Applicant's above-mentioned patent. 
Other realizations are, however, also possible. 
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The arrangement can advantageously be changed 
between at least a first function mode, for example the 
reading pen or mouse function, and a second command mode, 
in which the signal -processing means are arranged to car- 
ry out said identification of predetermined information 
in the recorded text and control of the electronic 
device. This embodiment has the advantage that it, simpli- 
fies the arrangement, as it only needs to interpret text 
in command mode. The danger of text which is only intend- 
ed to be entered and stored in the electronic device 
being wrongly interpreted as a command is also avoided. 

■As an alternative, the signal-processing means 
should be able to interpret continuously the content 
in the text recorded by the optical sensor. In order to 
avoid words being interpreted as a command which are only 
intended to be entered and stored in the electronic 
device but which consist of the same character combina- 
tion as a command, additional requirements can be made 
regarding what the signal-processing means are to inter- 
pret as a command. For example, it can be required that 
commands are to consist of characters in a particular 
font, font size or the like. 

in one embodiment the signal -processing means are at 
least partially located in the same casing as the elec- 
tronic device. By this means the input unit can be made 
simpler and cheaper. In addition the processor power 
which is already available in the electronic device can 



be utilized in order to carry out the functions of the 
signal -processing means. Certain processing of the 
recorded text can, however, advantageously be carried 
out in the input unit, for example localization of the 
text in the image or images and conversion of the text 
to character-coded format, for example ASCII code, so . 
that a smaller amount of information needs to be trans- 
mitted from the input unit to the electronic device. 

in a less preferred embodiment the signal-processing 
means can be completely integrated with the input unit so 
that the electronic device receives one or more commands 
directly from the input unit. In another less preferred 
embodiment the input unit can just record images and 
transfer these to the signal -processing means which car- 
ry out all the processing of the images. 

The input unit is advantageously arranged to commu- 
nicate with the electronic device by wireless means, so 
that the use of the input unit is as flexible as possible 
and so that certain functions can be used stand-alone. 
Alternatively, communication via a cable could be pos- 
sible, for example across a USB. 

in a particularly preferred embodiment the signal - 
processing means are arranged in such a way that when 
they identify an address for electronic mail in the 
. recorded text, the electronic device is caused to open 
a program for electronic mail. The arrangement preferably 
causes the device not just to open the program but also 



7 



to open a template for electronic mail. It is even more 
preferable for the template to be opened with the record- 
ed e-mail address already entered in the address field. 

The identification of the e-mail address can, ,for 
example, be carried out by recognition of @, the signal - 
processing means interpreting all the characters which 
are associated with ® as part of the address. 

In addition the signal-processing means can advan- 
tageously be arranged so that when they identify a web 
address in the recorded text the electronic device is 
caused to open a web search program. The device is pre- 
ferably caused not just to open the program but also the 
web page corresponding to the web address. The identifi- 
cation of the web address by the signal -processing means 
can, for example, be based on recognition of the charac- 
ter combination "http://" or "www", the signal -processing 
means interpreting all the characters which are associat- 
ed with said character combination as part of the web 
address. In this way the user can simply and quickly open 
a web page by using the same input unit he or she uses 
for other input unit functions. 

In addition the signal-processing means can advanta- 
geously be arranged so that when they identify a tele- 
phone number in the recorded text, the electronic device 
is caused to ring the telephone number. 

As mentioned above, the signal -processing means can 
create a predetermined command upon recognition of one or 
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re predetermined characters or words in the recorded 
text. The predetermined words can quite simply be desig- 
nation of the commands which are created. For this pur- 
pose the arrangement can advantageously comprise a pro- 
duct by means of which a number of command indications 
are shown. The command indications can advantageously be 
represented by character combinations which are easy for 
the user to understand. The product can, for example, be 
a mouse pad. 

According to a second aspect of the invent ion, this 
concerns a method for controlling an electronic device, 
comprising the steps of recording text by means of a 
hand-held input unit by imaging this by means of at 
least one image, identifying predetermined information 
in the imaged text, and controlling the electronic device 
depending upon the imaged text. 

The advantages of this method are apparent from the 
discussion of the arrangement. Where appropriate, charac- 
teristics of the arrangement also apply for the method.. 

in the Applicant's Swedish Patent Application No. 
9803455-6 an input unit is also known which has both an 
image-based mouse function and an image-based input func- 
tion. The function of the input unit is changed by the 
user pressing buttons. 

An additional object is to simplify the use of an 
input unit which has two functions so that the change 
between different functions can be carried out in a con- 
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venient way for the user. This object is achieved by 
means of a device according to claim 13. 

More specifically, according to a third aspect, the 
invention comprises an input unit which has at least a 
first and a second function. The input unit comprises 
image -capturing means, for example an optical sensor, for 
capturing images and image-processing means, for example 
a processor, for processing the images to achieve the 
above-mentioned two functions. The input unit is arranged 
to change from the first to the second function when the 
image-processing means detect predetermined information 

in one of said images. 

The predetermined information can in principle be 
any information which makes it possible for the input 
unit to interpret that it is to change from a current 
function to a new function. The information can, for 
example, consist of one or more predetermined characters, 
symbols, words, text in special font or line thickness or 
the like. When the input unit identifies the predetermin- 
ed information, it automatically changes to the required 
function, in this way the user does not need to press any 
buttons. 

In a preferred embodiment, the predetermined infor- 
■ mation is a predetermined pattern. If the input unit, for 
example, has a mouse function and another function, it 
can be programmed to be able to identify the pattern on 
a mouse pad. When the user places the input unit on the 
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mouse pad, the image -capturing means record an image 
of the pattern on the mouse pad. The image-processing 
means identify the pattern as predetermined information 
which indicates mouse function and the input unit changes 
automatically to mouse function and processes the images 
to achieve the mouse function. 

It is, of course, convenient if the input unit is 
also arranged to change from the second function to the 
first function when it detects other predetermined infor- 
mation. If the input unit has only a mouse function and 
an input function, it can, for example, change back from 
the mouse function to the input function when the image - 
processing meant detect that the predetermined pattern 
for the mouse function is no longer present in the cap- 
tured images. Alternatively, the change can be carried 
out on the basis of positive identification of another 
predetermined pattern. 

In an advantageous embodiment, the predetermined 

s 

information consists of a position- coding pattern, pre- 
ferably an absolute position-coding pattern. 

The advantage of position-coding patterns is that 
the predetermined information can consist of one or more 
specific positions. This makes it easier for the device 
to identify when it is to change, as it does not need to 
carry out any character recognition (OCR) . 

Absolute position-coding patterns are known for exam- 
ple from US 5,852,434 and the Applicant's SE 9901953-1 and 
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9903541-2 which were not publicly available at the time of 
the filing of the present application. 

The functions between which the change is made can, 
for example, be a mouse function, a scanner function, a 
handwriting/hand-drawing function or some, similar func- 
tion which can be carried out on the basis of captured 
images. 

Brief Description of the Drawing 

This invention will now be described in greater 
detail by means of one embodiment with reference to the 
accompanying drawing in which the figure shows schema- 
tically how an arrangement according to the invention can 
be constructed and used. 
Description of a Preferred Embodiment 

An embodiment of the arrangement is described below 
which comprises a mouse function, a scanner or reading 
pen function and a control function which is based on 
text recording. 

The figure shows a mouse pad 100, an electronic 
device 2 00 in the form of a computer and an input unit 
3 00 for the computer. 

The mouse pad has a working field 100 with an irre- 
gular pattern (not shown) which makes it possible to 
determine the relative positions of two images which have 
partially overlapping contents by means of the content of 
the images, and a command field 12 0, in which a number of 
predetermined commands are indicated. 
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The input unit 300 has a casing 1 in the shape of a 
pen. One short side of the casing has a window 2 through 
which images are captured for the different image -based 
functions of the input unit. 

The casing 1 contains principally an optics part, an 
electronics part and a power supply. 

The optics part comprises a number of light emitting 
diodes 6, a lens system 7 and an optical sensor 8 which 
constitutes the interface with the electronics part. 

The light emitting diodes 6 are intended to illumi- 
nate a surface which is at the moment below the window. 

The lens system 7 is intended to project an image 
of the surface which is below the window 2 onto the 
light-sensitive sensor 8 in as correct a way as possible. 
The optical sensor 8 can consist of a two-dimensional 
quadratic CCD unit (CCD = charge - coupled device) with a 
built-in A/D transducer. Such sensors are commercially 
available . 

The power supply for the input unit is obtained from 
a battery 12 . 

' The electronics part comprises a processor 20 with 
conventional associated circuits, such as various types 
of memory, and associated programs for carrying out the 
functions described here. The electronics part also 
comprises a transceiver 26 for transmitting information 
to/from the computer 200. The transceiver can be based on 
infrared technology or radio technology for transmission 
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over short distances, for example in accordance with the 
Bluetooth standard. The electronics part further com- 
prises buttons 27, by means of which the user can control 
the input unit and in- particular change the input unit 
between the mouse function, the scanner function and the 
control function. When the mouse function is being used 
the buttons can also have functions which correspond to 
the click buttons on a traditional mouse. 

The computer 200 is an ordinary personal computer 
with circuits and programs which make possible communi- 
cation with the input unit 300. However, in this embodi- 
ment this also contains signal -processing means which 
constitute part of the arrangement for controlling its 
function. The signal -processing means consist of a pro- 
gram which is installed in the computer. This is shown 
symbolically by broken lines and reference numeral 210. 

As mentioned, the input unit 3 00 has a scanner func- 
tion, a mouse function and a control function. 

The scanner function is used to record text. The 
user passes the input unit across the text which he wants 
to record. The optical sensor records images with par- 
tially overlapping contents. The images are put together 
by the processor 20. Each ■ character in the put-together 
image is localized and, using for example neural net- 
work software in the processor, it is determined to which 
ASCII character the character corresponds. The text con- 
verted in this way to character-coded format can be stor- 
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ed in the input unit or transferred to the computer 200. 
The scanner function is described in greater detail in 
the Applicant's Swedish Patent No. 9604008-4. 

The mouse function is used to control a cursor on 
the display 201 of a computer 200. The mouse function is 
also image-based in this embodiment. When the input unit 
300 is moved across the working field 110, the optical 
sensor 8 records a number of images with partially over- 
lapping images. The processor 20 determines positioning 
signals for the cursor of the computer 2 00 on the basis 
of the relative positions of the recorded images, which 
are determined by means of the contents of the images. 
The mouse function is described in greater detail in the 
Applicant's Swedish Patent No. 9803456-4. 

The control function is based on the scanner func- 
tion. The user records text in the same way as in the 
scanner function. The text is sent in character-coded 
format from the input unit's transceiver to the signal - 
processing means 210 in the computer 200, together with 
an indication that this is control information which is 
to be interpreted. The signal -processing means examine 
the received text and search for predetermined informa- 
tion in this in the form of predetermined characters and 
character combinations. When such predetermined informa- 
tion is found, the signal -processing means create pre- 
determined commands to the computer as a function of the 
predetermined information . 

15 



The arrangement described above is used in the fol- 
lowing way. First assume that the user wants to use the 
input unit as a mouse. He selects the mouse function by 
means of the buttons 27. By moving the input unit on the 
working field he controls the cursor on the display 201 
of the computer 200. Assume next that the user edits a 
document in the computer 200. He can then mark text by 
"clicking" with the buttons 27, Assume that the user 
first wants to replace a first piece of text with a 
second piece of text which is situated elsewhere in the 
text. The user marks the second piece of text using the 
input unit. Then he changes the input unit to the control 
function and records the command "cut" by passing the 
input unit across this command on the mouse pad 100. The 
input unit then sends the character-coded text "cut" to 
the signal -processing means 210 in the computer 200, 
which identify the text as a command and create a corre- 
sponding command for the word-processing application con- 
cerned, which cuts out the marked piece of text. The user 
next marks the first piece of text using the input unit 
and then causes the computer to paste the cut-out piece 
of text in place of the marked text by recording the com- 
mand "paste" using the input unit. 

Now assume that the user next wants to enter text 
from a newspaper in his document. He first positions the 
cursor in the required place using the mouse function 
of the input unit. Then he changes the input unit to the 
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scanner function and scans in the text from the news- 
paper. The text is converted to character-coded format 
and transmitted to the signal -processing means which 
insert the text in the position marked by the cursor. 

Now assume that the user sees an interesting web 
address in the newspaper he is reading and wants to look 
at this web page. He then changes the input unit to con- 
trol function and reads off the web address from the 
newspaper. The recorded text is transferred to the 
signal -processing means which identify the character com- 
bination "http://" and cause the computer to open the web 
page with the recorded address. 

Finally, assume that the user wants to send an 
e-mail to a friend. He uses the control function of the 
input unit to record the command "e-mail" on the mouse 
pad. Recognition of this command by the signal -processing 
means results in the means generating a command to the 
computer which causes it to open the e-mail program. The 
user can then record the required e-mail address and even 
the content of the message using the scanner function. 

As shown above, the user can conveniently carry out 
a number of different functions which comprise inputting 
information and controlling the computer 200 by means of 
j ust one input unit . 

Of course, other functions can be integrated into 
the input unit in order to further increase its usabili- 
ty. An example is a function to record handwritten text. 
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which is described in the Applicant's Swedish Patent No. 
xxxxxxx . 

other commands can be created in addition to those 
described above. A user can also himself define how 
recorded text is to be interpreted by the signal -process - 
ing means and in which control of the computer a particu- 
lar recorded piece of text is to result. 

The change described above between the different 
functions is carried out by the user pressing buttons 
on the input unit . 

As an alternative to this, the input unit can itself 
detect that it is to change between different functions. 

The input unit can, for example, be arranged to 
search for predetermined information in each image which 
is captured by the optical sensor 8. The predetermined 
information can, for example, be the pattern on the work- 
ing field 120 of the mouse pad. When the processor 20 
detects this pattern, it changes to the mouse function 
and processes the images in the way described above to 
provide positioning signals for the cursor on the display 
201 of the computer 200. When the user thereafter places 
the input unit on a newspaper in order to scan in text, 
the processor 2 0 no longer detects the mouse pad pattern 
and so it knows that it is to change to the scanner func- 
tion and process the images in the way described above 
for the identification of text and conversion of imaged 
text to character-coded format. This technique can also 
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be used for changing to the control function. The com- 
mands in the mouse pad's command field can be written in 
a particular way so that the processor can detect that 
the characters are not characters which are to be entered 
into the computer by the scanner function, but are char- 
acters which represent commands and are to be sent to the 
signal -processing means in order to be processed as such. 
The commands can, for example, be written in a particular 
size, particular font or particular line thickness. 

As another example, the change can be implemented 
on the basis of change commands which are written in the 
command field of a mouse pad. When the user wants to 
change to scanner function he records the word "scanner'' 
from the command field using the input unit. The proces- 
sor 20 identifies this as predetermined information which 
indicates that it is now to carry out a scanner function. 

Above the pattern on the mouse pad 100 is described 
as an irregular pattern and the mouse function is 
described as being achieved by determination of the 
relative position of the recorded images. In another 
embodiment, the pattern on the mouse pad can be a posi- 
tion-coding pattern, which systematically codes positions 
over the whole of the mouse pad. In this case the mouse 
function can be based on reading off positions using the 
position-coding pattern. In addition the change to mouse 
function can be based on recognition of the position- 
coding pattern. In addition particular positions or posi- 
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tion areas (also called domains) can be dedicated to par- 
ticular functions, for example those corresponding to the 
different commands in the command field. When the proces- 
sor detects a particular position, it determines which 
function corresponds to this position. In this way the 
input unit can be caused to change from one function to 
another function by placing it in a particular position 
on the mouse pad. Different domains of the position- 
coding pattern can also be dedicated to commands for 
controlling the computer 200. Instead of the signal- 
processing means in the computer 200 detecting predeter- 
mined information in the text which is entered, they can 
thus detect positions in the form of coordinates and 
identify which command is to be created to control the 
computer 200. For example, if the user wants to open the 
e-mail program in the computer 2 00, he can place the 
input unit on the mouse pad in a position where it says 
"e-mail". The optical sensor 8 records an image of the 
position-coding pattern in this position. The processor 
identifies which position, that is which coordinates, 
corresponds to the position-coding pattern in the image. 
It sends the coordinates to the signal -processing means 
in the computer 200. The signal -processing means identi- 
fy that these coordinates mean that they are to create 
a command to the computer which causes it to open the 
e-mail program. 
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The mouse pad can thus be divided into position 
areas or domains with different functions. An additional 
example of this is that one area can be dedicated to 
relative mouse function (the cursor is moved in the same 
way as the input unit) and another area to absolute mouse 
function (the cursor is placed in the position which cor- 
responds to the position of the input unit on the mouse 
pad) . The input unit itself understands which function 
it is to use on the basis of whether the position-coding 
pattern (and hence the identified coordinates) belongs to 
one or the other area. Of course, alternatively the same 
surface can be used for the relative and absolute func- 
tions and the change can be carried out by means of 
change commands in the way described above.' 

further example is that the mouse pad can have 
an area which is dedicated to a scrolling function. The 
input unit can thus be a mouse with various mouse func- 
tions. It can also be a mouse which, in addition to con- 
trolling a cursor on a computer, can control other func- 
tions of the computer or other electronic devices, such 
as mobile telephones or PDA. 

The above example is just one example of how the 
arrangement according to the invention can be designed. 
Based on the summary of the invention, experts in the 
field can achieve a number of variants of this example. 
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What we claim and desire to secure by Letters Patent 

is : 

1. An arrangement for controlling an electronic 
device, characterized by an input unit with 
an optical sensor for recording text by imaging this by 
means of at least one image, and signal -processing means 
for identifying predetermined information in th'e imaged 
text and for controlling the electronic device dependent 
upon said predetermined information. 

2. An arrangement according to claim 1, wherein the 
predetermined information consists of at least one pre- 
determined character and wherein the signal -processing 
means are arranged to generate a command associated with 
said character for controlling the electronic device. 

3. An arrangement according to claim 1 or 2 , which 
arrangement comprises a reading pen function. 

4. An arrangement according to any one of the pre- 
ceding claims, which arrangement comprises an optical 
mouse function for controlling a cursor on a display on 
the electronic device. 

5. An arrangement according to any one of the pre- 
ceding claims, which arrangement can be changed between 
at least a first function mode and a command mode, in 
which the signal -processing means are arranged to carry 
out said identification of predetermined information in 
the recorded text and control of the electronic device. 
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6. An arrangement according to any one of the pre- 
ceding claims, in which the signal -processing means are 
at least partially located in the same casing as the 
electronic device . 

7. An arrangement according to any one of the pre- 
ceding claims, wherein the input unit is arranged to com- 
municate with the electronic device by wireless means. 

8. An arrangement according to any one of the pre- 
ceding claims, wherein the signal -processing means are 
arranged in such a way that when they identify an address 
for electronic mail in the recorded text, they cause the 
electronic device to open a program for electronic mail. 

9. An arrangement according to any one of the pre- 
ceding claims, wherein the signal -processing means are 
arranged in such a way that when they identify a web 
address in the recorded text, they cause the electronic 
'device to open a web search program. 

10. An arrangement according to any one of the pre- 
ceding claims, wherein the signal -processing means are 
arranged in such a way that when they identify a tele- 
phone number in the recorded text, they cause the elec- 
tronic device to phone the telephone number. 

11. An arrangement according to any one of the pre- 
ceding claims, comprising a product on which a number of 
command words are indicated. 

12. A method for controlling an electronic device, 
comprising the steps: 
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recording text by means of a hand-held input unit, 
by imaging this by means of at least one image, 

identifying predetermined information in the imaged 
text, and 

controlling the electronic device dependent upon the 
imaged text. 

13. An input unit with at least a first and a second 
function, comprising image -capturing means for capturing 
images and image-processing means for processing the 
images to achieve said two functions, character- 
ized in that the input unit is arranged to change from 
the first to the second function when the image-process- 
ing means detect predetermined information in one of said 
images . 

14. An input unit according to claim 13, wherein 
said predetermined information is a predetermined pat- 
tern . 

15. An input unit according to claim 13 or 14, 
wherein the input unit is arranged to change from the 
second function to the first function when it detects 
other predetermined information. 

16. An input unit according to any one of claims 
13-15, wherein said predetermined information consists of 
a position-coding pattern, preferably an absolute 
position-coding pattern. 
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17. An input unit according to any one of claims 
13-16, wherein the first function is a mouse function 
and the second function is an input function, preferably 
a scanner function. 



Abstract of the Disclosure 



An arrangement for controlling an electronic device 
has an input unit with an optical sensor for recording 
text by imaging this using at least one image, and 
signal -processing means for identifying predetermined 
information in the imaged text and for controlling the 
electronic device dependent upon said predetermined 
information . 
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